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ABSTRACT 

The place and importance of implementation in the 
life cycle of Army training programs is frequently mi sunder stood « 
Typically, a program's life cycle is thought of as research, 
development, and use. If implementation is thought of at all, it is 
regarded as an event, not a process. Many worthwhile programs have 
failed because the implementation process was neglected. 
Impleme^ntation should be regarded as a high-risk period in a 
program's life cycle. The implementation stage must be viewed as an 
organizational research and development stage in which the user 
decides where a program fits into the organization's priorities and 
the developer and user work together to plot a mutual accommodation 
of the program to the organization and the organization to the 
program. Planning for implementation of a training program involves 
deciding what actions are required to field, sustain, and support the 
program; what agencies should have responsibility for these actions; 
and how the cooperation of these agencies can be obtained. The 
analysis-of-f it model that has been developed to facilitate the 
accommodation process calls for fidelity and sufficiency evaluations 
to discover changes that are required in the following aspects of 
training program implementation: organizational arrangements, 
individual know-how, organizational rules, and individual commitment. 
The second phase of the model includes the following strategies for 
accomplishing the program's prerequisite goals: assistance, 
education, power, and persuasion. (MN) 
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FOREWORD 



"To make the future happen sooner" it is not enough to develop or buy 
state-of-the-art training programs. These programs have to be aggressively 
integrated into the users' training environment. 

As the Army's major behavioral science research and development agency, ARI 
has been involved in a number of programs that looked good to the researchers 
and developers (for example. Training and Doctrine Command (tRADOC), Development 
and Readiness Command (DARCOM), U.S. Amjy Forces Command (FORSCOM), U.S Ariny 
Europe (USAREUR)) but vere simply not used by their target audience (for exam- 
ple, active Army units or TRADOC schools). This failure to transfer technology 
is distressing, and recently ARI has launched an important effort to find out 
why some veil-designed programs succeed while equally promising ones fail. 

Much of our research points to implementation as a key but neglected stage 
in a program's life cycle. This report provides an overview of implementation 
issues. Its main thesis is that users, developers, and researchers all have a 
stake in implementation, all have a unique role to play, and all can gain from 
a better understanding of the issues that implementation raises. The stakes 
are high. More attention paid to the process of implementation will result in 
the more effective use of training programs and increased readiness. 




EDGAR M. JOHNSON 
Technical Director 



PREFACE 



While there is a recognized need for new and better ways to train 
soldiers to fight, many training programs developed in response to this 
need are used poorly or not at all. Many of these programs fail to be 
implemented while others are so changed by the user that the program £S 
used bears only a nominal relationship to the program that was 
fielded. To begin solving Implementation and use problens, these 
problems must be viewed in the context of the program's life cycle. In 
this paper, the life of a graining program is divided into three major 
sets of issues: research and development, implementation, and use. 

It is argued that Implementation is a distinct stage in a program's 
life cycle which involves "organizational research & development." 
During this stage both the developer and user have dlsMnct, but 
mutually supportive, responsibilities. The user must decide (with 
developer input) where the program fits into the organization's 
priorities and how many resources can be devoted to it. The developer 
must decide how to make changes and compromises in the program so that 
its maximum value can be achieved with the resources available. 



IMPLEMENTING ARMY TRAINING PROGRAMS: AN OVERVIEW FOR MANAGERS 



EXECUTIVE SUMMARY 



Requirement : 

The place and importance of implementation in the life cycle of Army train- 
ing progrms is not understood, lypically, a program's life cycle is thought of 
as research, development, and use: if implementation is thought of at all, it 
is regarded as an event, not a process. Unfortunately, many worthwhile programs 
have failed because the implementation process was neglected. 



Procedure: 

The view adopted here is that implementation is a high-risk period in a 
program's life cycle. The cost of ignoring implementation is measured by wasted 
research and development dollars and uissed opportunities to improve training. 
The benefit of planning and monitoring implementation is the moro effective use 
of training programs and increased readiness. 



Findings : 

An overview of implementation issues is presented. Inrplementation is 
viewed as an "organizational research & development" stage. In this stage the 
user decides where a new program fits into the organization's priorities. The 
developer and user then work together to plot a "mutual accommodation" of the 
program to the organization and the organization to the program. 



Utilization of Findings: 

The Report is intended for developers and users of Army training programs 
(for example, TRADOC Program Managers, ARI Team Leaders^ private contractors. 
Army schools, and operational units). The intent is to identify for Army de- 
velopers and users the issues involved in implementation. The hope is that a 
better understanding of implementation vill result in more time and attention 
paid to implementation. The ultimate goal is to increase the effectiveness of 
training programs for a better trained Army. 

Readers already convinced of the value of iTq)lementation should refer to 
the following: 

A Guide to Implementation of Training Products (ARI Technical Report 1350) 
for a "users guide" to planning an implementation; and 

Implementation Monitoring; A Role for Evaluators in Helping Innovations 
Succeed (ARI Technical Report, in press) ^ for a technical discussion of 
the issues and procedures involved in monitoring an ongoing implementation 
effort. 
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IMPLEMENTING ARMY TRAINING PROGRAMS: 
AN OVERVIEW FOR MANAGERS 



Army programs face many of the same problems as programs in education 
or industry. Programs developed in response to real needs fail to be imple- 
mented and most of those that are implemented are modified and used quite 
differently than intended by the program's developer. 

The Amy Research Institute has had a continuing interest in the imple- 
mentation and use of Amy training programs. We have recently developed 
a threefold approach to implementation. 

• Initiate case studies of the problems vhich training programs face 
and must overcome if they are to be successfully implemented and 
used. 

• Provide guidance vhich Army sponsors can use to plan the implemen- 
tation of new training programs (T. Gray, Roberts-Gray, & W. Gray, 
1983). 

• Develop a framework for monitoring and evaluating the implementa- 
tion of training programs (W. Gray, 198U). Such a framework starts 
with the process of iii5)lementation and continues, ideally, until 
the program either fails or, if successful, becomes obsolete. 

In this paper I present an overview of important implementation issues. 
The intended audience is those managers who are either about to hand-off a 
new program (researchers or developers) or about to implement one in their 
unit. Those desiring a more technical discussion of implementation are 
referred to W. Gray, 198U. 
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IMPLEMENTATION 
FROM THE 
DEVELOPER'S 
POINT OF VIEW 




PROUD 
DEVELOPER 



PRODUCT 

MANAGER NEEDY USER 



IMPLEMENTATION 
FROM THE 

PRODUCT MANAGER'S 
POINT OF VIEW 




ABSENT OVERBURDENED UNINTERESTED 

DEVELOPER PRODUCT MANAGER USER 



IMPLEMENTATION 
FROM THE USER'S 
POINT OF VIEW 




JUST ANOTHER PRODUCT 



TYPICAL ARMY 
TRAINING ENVIRONMENT 



FIGURE 1: Points of View on New Training Programs 

(Reprinted from ARI Technical Report 1350, September 1983) 
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Better Mousetrap or Alligator Farm? Obstacles to Implementation 



The basic problem with the implementation process is the lack of an 
implement or, that is, neither the developer nor the user is willing to 
invest the time and energy required to properly implement a new program 
into the user's organization, (See figure 1.) The developer has a BETTER 
MOUSETRAP which he or she expects to be greeted with a round of applause 
from eager users. Unfortunately, the intended users are already busy doing 
other things and are likely to perceive the program more as an alligator 
than as a golden opportiinity. 

Identify Crisis or the Name is Familiar But I can't Place the Fa.ce 

If a training progreun survives implementation and is used routinely, 
maximum return on the investment is still not assured*. The problem with 
use is that the program as used is seldom, and maybe never, identical to 
the program that was developed. If the only information on the effective- 
ness of a program comes from the research and development stage, then very 
little is known about the effectiveness of the program as used by an 
organization. 

Changes in the program by the user are a fact of life. For example, 
a Rand project that looked at the implementation of 293 educational proj- 
ects found NO cases in which the project was implemented unchanged (Berman, 
1978). All 293 projects were either not implemented at all, or if imple- 
mented, had been changed by the user. 

At this point, I hope you (the reader) are beginning to be convinced 
of two things. First, new training programs do not Just get used. They 
must be integrated aggressively into the user organization (W. Gray, 1982). 
Second, an implemented program is always different from the program the 
training developer produced. 
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Organizational Research & Development 

In what follows. Implementation is treated as an "organizational 
research and development" process. This view contrasts with the notion 
that Implementation consists of the developer handing off a finished 
product which the user simply plugs in to solve a problem. 

In organizational R&D the basic question for the user is — how 
Important is the new program? Where does it fit into my hierarchy of 
organizational needs and goals? (The developer, through years of narrow 
focus, often regards his/her newest program as solving the most 
important of the Army's problems. On the positive side, this attitude 
leads to hard work and high quality products. On the negative side, it 
can resiilt in programs which require more resources than the user can 
provide or that the problem objectively deserves.) After the program's 
importance is decided, organizational R&D becomes a process of mutual 
accomodation of the program to the unit and the unit to the program. 
The goal is to maximize program effectiveness within the constraints 
(resources, time, effort, other priorities^ and so on) provided. 



Factors Affecting the Implementation & Use 
of New Training Programs 
The problems which arise during the implementation and initial use 
of new training programs can be handled (if not always anticipated) by 
proper planning and careful monitoring. Implementation is best viewed 
as a dynamic process not a fixed event. Problems vary as a function of 
where the program is in its life cycle. Some problems arise, and are 
best solved, before the program becomes well established. Other 
problems emerge only after the program has been used for some period of 
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time. To best understand what problems occur when, an understanding of 
the program's life cycle (from an implementation perspective) is 
required (see figure 2). 



Figure 2 shows four classes of R&D issues which influence 
implementation and use. First is the condition analysis. This may be a 
broad look at all of Army training or narrower look at some area that is 
thought to be a problem* Out of the condition analysis comes a problem 
statement. As an example, the Army in the early 70's concluded that 
small-unit tactical training uas in need of improvement. At this point, 
the training research community was called upon to develop a solution 
concept. To improve small-unit tactical training, a two-part solution 
was sought. One, develop a new team training methodology; two, develop 



turned over to the developers for solution development. The developed 
solution was then measured against the concept in the Amy's development 
test/operational teet (DT/OT) cycle. 

For most training programs, involvement of the R&D community ends 
at this point and the program is turned over to the user to do with as 
s/he pleases. However, there are important reasons for the developer to 
actively help implement the program and evaluate its use. As shown in 
figure 2 (by the line from "Effectiveness Evaluation" to "Problem 
Statement"), the real test of the developer's product is in its use in 
the field: Does use of the new program solve the training problem? For 
the state of field training to be a valid measure of a program's 
effectiveness requires that the program be implemented with an 
"acceptable" level of fidelity. Hence the developer has a vested 



Research & Development Issues 




This two-part concept was 
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IMPLEMENTATION ISSUES 
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FIGURE 2: The Trainmg Program Life Cycle from an 
Implementation Perspective 
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Interest in working with the user to ensure that implementation problems 
are identified and overcome* 

(REALTRAIN is an excellent example (see Scott, 1983). Controlled 
studies showed dramatic ijnprovements in tactical proficiency among 
REALTRAIN trained troops (for example. Bank, Hardy, Scott, Kress, & 
Word, 1977), yet Implementation problems were never resolved. Despite a 
worldwide fielding effort by the developer, REALTRAIN never obtained 
wide-spread use and quickly died a quiet death (Roberts-Gray, Clovis, 
T.Gray, Muller, & Cunningham, 1981).) 
Implementation Issues 

Implementation issues are tb<sse plans and actions required to 
aggressively integrate the new program into the operational 
environment. Planning an Implementation involves decidipg: (a) what 
actions are required to field and to sustain and support the program; 
(b) what agencies should have responsibility for these actions; and (c) 
obtaining the cooperation of these agenciis. 

Figure 2 depicts the three issxies which should be monitored duripg 
implementation* An Implementation plan can be considered as a set of 
"planned actions". The manager should determine whether the plan 
contains all actions necessary for Implementing the program and any that 
are unnecessary. When a planned action is executed then a product of 
that plan exists. The manager should know whether the product achieved 
the goals planned for it or whether something was lost duripg the 
execution. For example, many Army training programs require new 
equipment and require that the trainer be able to perform some low level 
maintenance on the equipment. A "planned action" might be the 
production of a pamphlet for the trainer on how to trouble shoot the 
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equipment. The particular pamphlet that is produced is a product of 
this planned action. We can then ask whether this pamphlet provides all 
the information needed to troubleshoot the equipment and whether the 
reading level and format is appropriate for its intended audience. 

The Important point here is that both plans and products have to be 
good if the new program is to be successful. Too often plans are 
carefully made but during their execution a checklist mentality 
prevails. That is, at execution, product quality is not measured, plan 
accomplishment is. The result is that any product, no matter how poorly 
done, enables a planned action to be checked off as accomplished. 

The ultimate goal of Implementation plans is to get the new program 
used routinely. However, evaluation of routine use typically takes 
place after most Implementation activity has ceased. Therefore, if we 
want to monitor "likelihood" of routine use we have to assess whether 
the Implementation process is achieving certain pre^requisit^ goals . 

The idea of pre-requisite goals must be elaborated even in an 
overview paper. For a training program, such as MILES (multiple 
integrated laser engagement simulation) to be successful, certain pre- 
requisite goals must be met. For example, MILES trainers (NCOs) must be 
able to diagnose and troubleshoot certain equipment malfunctions. 
Teaching trainers how to troubleshoot MILES equipment Is a pre-requisite 
goal of the Implementation program. As another example, for any new 
program to be used, a certain amount of organizational inertia (and 
resistance) must be overcome. Some of the implementation plaTis must be 
directed at overcoming this inertia. (In the example of MILES, this 
inertia was overcome through a combination of command emphasis, new 
rules and r^ulations regarding tactical training, and demonstrations 
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which emphasized MILES' realism*) 

The Place of Theory » An adequate theory of implementation would 
serve to Identify certain classes of potential problems and suggest 
strategies for overcoming these problems. ARI has sponsored the 
development of a model which works for most Army training programs 
(Roberts-Gray & T.Gray, 1983). The model (see Figure 3) provides a 
basis for analyzing the fit between the iiinovation and user. With this 
information^ the model yields an analysis of charges in organizational 
arrangements, individual know-how, organization rules, and individual 
commitment that are required if the innovation is to "fit" the user. 
These changes become the pre-requisite goals of the Implementation 
process. Finally, for each change, the model yields a suggested 
strategy for accomplishing that change. 
Use Issues 

As mentioned eailiei the program as used is seldom identical to 
the program that was developed. Hence, it is necessary to describe the 
program that is actually used and to assess its actual, as opposed to 
theoretical, effectiveness. From au Implementation perspective, these 
concerns can be organized into the categories of fidelity, sufficiency, 
and effectiveness (see Figure 2). Each category of use issues is 
related to a category of R&D issues as well as being interrelated with 
the other use issues. 

Fidelity . Fidelity evaluation (Fullan & Pomfret, 1977) is 
procedure oriented. It is a comparison of the user's procedures against 
the developer's ideal. The goals of the fidelity evaluation are to 
determine what parts of the program are actually used and to describe 
variations in use among different users. The data from the fidelity 
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Analysis-of-Fit 

Between the Innovation & User 

Pie-Requisite Goals 

Changes Required in: 

• Organizational Arrangements 

• Individual Know-How 

• Organizational Rules 

• Individual Committment 
For the Innovation to "Fit" the User 



Strategies 

For Accomplishing Fre-Requisite Goals: 

• Assistance 

• Education 

• Power 

• Persuasion 

FIGURE 3: A Model of Implementation 
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evaluation provides feedback to the developers on how well their product 
is used and feedback to the implementors on the Implementation 
process. Results of the fidelity evaluation may lead Ijnplementors to 
launch a second, revised effort at implemeatation. 

Sufficiency. The sufficiency evaluation is function oriented. It 
compares a user's practice against an ideal model of "how-to-train." 
(The area of sufficiency evaluation has been referred to by Leinhart 
(1980) as Domain-of-Inst ruction.) We assume that in the solution 
concept stage of R&D, the researchers had an implicit theory of what 
functions the trainer must perform to conduct good training (such as 
that provided for teacher functions by Fisher, Berliner, Filby, 
Marliave, Cahen, & Dishaw (1981)). Each function was then instantiated 
during solution development to form the exact procedures which define 
the particular program. Since we know that users always change a 
program by dropping, adding, or altering procedures, it is more than 
possible that a function is being fulfilled by procedures different than 
those the developer provided. This situation is what sufficiency 
evaluation is designed to assess. For example, many training programs 
include procedures which function to provide feedback* to the trainees. 
However, if the exact procedures specified by the program are not 
followed, feedback may still be provided by some other procedures. 
Hence, we could find the case where excellent feedback is being provided 
but the procedures railed out by the training program are not 
followed. That is, the function is being filled, but the procedures are 
not . allowed . 

Sufficiency evaluation is important because it gets us away from 
the assumption that any change in the program is bad. If the users 
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chaoge the program to brlpg it more in line with their way of doipg 
things, then the users may have substituted procedures of their own 
which fill the same function as the procedures invented by the 
developer* 

Effectiveness * The effectiveness evalxiation should be (but usually 
is not) a "user oriented" comparison of the current state of training 
with the pre-fieldii^ state of training. It is not an experiment. The 
purpose is not to assess the "maximum" effectiveness of the system, but 
to assess its actual effectiveness when used routinely by real users. 
The goal of this evaluation is to decide whether the problem which led 
to the development and fielding of the program has been solved. 

Conclusions & Perspectives 

Experience with Army training programs has led ARI to the belief 
that attention to the process of Implementation is vital if a program is 
to become a routine part of unit training. ARI has developed guidance 
for implementation planners (T.Gray, Roberts-~Gray, & W»Gray, 1983) and a 
framework for Implementation monitoring (W.Gray, 1984). The framework 
is Army oriented. It organizes the monitoring issues in terms and 
categories attuned to the political realities and trainit^ issues with 
which the Army user is familiar. 

Probably the biggest implementation problem is the lack of an 
implementor. The user is typically overburdened (see figure 1) with 
routine tasks and has no resources to spend assessing the impact a new 
program will have on his/her plans, procedures, resource requirements, 
and so on. The developer^ s mission is to develop and maybe deliver the 
new program. For the developer a delivered program is a dead issue for 
which s/he has neither the time nor resources to track. 

ERIC 



Monitoring Impleraentatlon can create oi^ganlzatlonal conflicts. 
Developer and user organizations are often jeolous of each other's 
authority, xhe fine line between evaluating the usability of a 
program and evaluating how a unit uses a program (that Is, evaluating 
the program versus evaluating the unit) Is hard to draw. Furthermore, 
both users and developers understand that In the real-world (as opposed 
to the R&D laboratory) failure tends to precede blame. If a program 
falls the reasons for that failure are better left unprobed rather than 
being attributed to a failure of usability or a failure of use. 

The way out of this dilemma is to view implementation as a stage in 
the program's life cycle during which the developer and user have 
distinct but mutually supportive roles. For both the issue is how to 
get maximum training value from the program in the context of the user's 
training environment. For the developer the focus is on how to modify 
the program so that it better fits that environment. For the user the 
task is to modify the environment to best support the program (without 
sacrificing other equally or more important programs). The goal is to 
treat implementation as a stage during which "organizational research 
and development" is needed to best decide where the program fits in with 
the organization's priorities and, given the resources available, how it 
can be used to maximiim advantage. 
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